Skip to content

Add optional PostgreSQL native materializer#61

Open
sam-saffron-jarvis wants to merge 1 commit into
discourse:mainfrom
sam-saffron-jarvis:pg-native-materializer
Open

Add optional PostgreSQL native materializer#61
sam-saffron-jarvis wants to merge 1 commit into
discourse:mainfrom
sam-saffron-jarvis:pg-native-materializer

Conversation

@sam-saffron-jarvis
Copy link
Copy Markdown

@sam-saffron-jarvis sam-saffron-jarvis commented May 29, 2026

What

Adds an optional PostgreSQL-only native materializer packaged as the companion gem mini_sql-pg_native.

Install path:

gem "mini_sql"
gem "mini_sql-pg_native"

The PostgreSQL deserializer attempts to load mini_sql/pg_native if present. If the companion gem is not installed, behavior is unchanged. This keeps require "mini_sql" from loading pg for non-PostgreSQL users. The native path can be disabled with MINI_SQL_PG_NATIVE=0.

The PR also updates CI from actions/cache@v2/checkout@v2 to v4, because GitHub now hard-fails the old cache action.

The native extension accelerates MiniSql::Postgres::Connection#query and #query_each row object materialization by reusing pg's own result type map and setting row instance variables directly from C. Other adapters are unchanged.

Notes

  • This is PostgreSQL-only; MySQL, SQLite, and JRuby paths are unaffected.
  • query_decorator intentionally falls back to the Ruby materializer to preserve decorator behavior.
  • The optional native extension is loaded lazily when the PostgreSQL deserializer is loaded, not from top-level mini_sql.
  • The companion gem is kept in this repo but excluded from the base mini_sql gem package. The Gemfile/Rakefile now specify mini_sql explicitly because Bundler sees two gemspecs in the repo.
  • The native extension depends on pg 1.6 internals (pg.h, t_pg_result, t_typemap), so the companion gem constrains pg to >= 1.6, < 1.7.
  • query_each/query_each_hash now initialize the PostgreSQL type map before entering single-row mode; native testing exposed that lazy initialization after send_query can conflict with a busy connection.

Benchmarks

Short local benchmark, Ruby 3.4.8, pg 1.6.3, PostgreSQL 18.4, local socket.

Simple materialization-only benchmark:

native:
  simple_mat_1k_2col      7.870k i/s  (127.07 μs/i)
  simple_mat_1k_6col      3.599k i/s  (277.83 μs/i)
 simple_mat_10k_6col    332.245 i/s  (3.01 ms/i)

ruby:
  simple_mat_1k_2col      5.846k i/s  (171.05 μs/i)
  simple_mat_1k_6col      3.078k i/s  (324.89 μs/i)
 simple_mat_10k_6col    294.499 i/s  (3.40 ms/i)

Approximate gain:

  • 1k rows / 2 cols: ~35%
  • 1k rows / 6 cols: ~17%
  • 10k rows / 6 cols: ~13%

Earlier full-query benchmarks showed ~5-22% gains depending on row width/type mix; wider typed rows are increasingly dominated by pg type decoding.

Testing

Passed:

ruby -Ilib test/mini_sql/postgres/native_materializer_test.rb
MINI_SQL_PG_NATIVE=0 ruby -Ilib test/mini_sql/postgres/native_materializer_test.rb
ruby -Ilib:test test/mini_sql/postgres/connection_test.rb
ruby -Ilib -c lib/mini_sql.rb
ruby -Ilib -c lib/mini_sql/postgres/deserializer_cache.rb
ruby -Ilib -c lib/mini_sql/postgres/connection.rb
ruby -Ilib -c lib/mini_sql/pg_native.rb
ruby -Ilib -c mini_sql.gemspec
ruby -Ilib -c mini_sql-pg_native.gemspec
git diff --check

Packaging smoke:

gem build mini_sql-pg_native.gemspec
gem build mini_sql.gemspec
gem install ./mini_sql-1.6.0.gem ./mini_sql-pg_native-1.6.0.gem --local --no-document --ignore-dependencies
ruby -e 'require "mini_sql"; p defined?(MiniSql::Postgres::Native::RowMaterializer)'

Output:

"constant"

Full rake test was attempted but this local container is not a clean mini_sql test environment: MySQL server is unavailable, and sqlite3 2.9.4 causes existing SQLite failures where the gemspec asks for sqlite3 ~> 1.4.4. The PostgreSQL tests above passed.

@sam-saffron-jarvis sam-saffron-jarvis force-pushed the pg-native-materializer branch 4 times, most recently from 9980ca6 to 7023c4b Compare May 30, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant